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Develcpraent of the SPRE Compliance Matrix for NMSU-A 



Discussion 



The New Mexico Commission on Higher Education issued Final Draft Standards for 
SPRE in September 1994. Smce then, NMSU-Alamogordo, a 2300 headcount branch 
community college of New Mexico State University has actively developed SPRE implementation 
strategies. We feel that the three-year-old Integrated Assessment and Strategic Planning (lASP) 
process is the best vehicle to comply vnlh pending SPRE standards. lASP was developed during 
our NCA self-study. Presently, we are faced vAth multiple requirements of the Student Right-to- 
Know and Campus Security Act, the 1992 amendments to the Higher Education Act, the Carl D. 
Perkins Vocational and Applied Technology Education Amendments of 1990 (Public Law 101- 
392), Title IV recertification requirements, and other external sources. These requirements and 
our own desire to assess and improve program offerings and student learning, propel our 
continued assessment efforts. The operation of lASP is described m the 1994 edition of .4 
Collection of Papers on Self-Study and Institutional Improvement^ **Using the Self-Study and 
Institutional Assessment to Facilitate Positive Institutional Change^* (Leas & LilUbridge, 
1994). The approach we used to modify L\SP to meet changing needs at the college and the 
progress we have made using lASP to comply with SPRE and other accountabihty standards is 
described in SPRE and the NMSU-A Integrated Assessment and Strategic Planning (lASP) 
Process: What We've Learned and Where We're G<?mg (Twomey, LilUbridge, Hawkins, & 
Reidlinger, 1995). This paper reports how we actually developed the SPRE Compliance Matrix. 
The SPRE Compliance Matrix was designed to help us determine if federal and state 
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accountability requirements are consistent with our mission and purposes. Because a major part 
of the SPRE standards deals with vocational programs, a small portion of our Carl Perkins grant 
served as the primary source of funding for the outcomes assessment project. 
Motivation 

New Mexico State University-Alamogordo is located in south central New Mexico. 
Alamogordo has a population base of 38,000 that is diverse and well-educated. Local military 
installations drive the economy. Because Alamogordo is distant from a major population area, 
many institutions m Otero County, includmg NMSU-Alamogordo, have chosen to lead the effort 
m doing things 'Yight". One of the strengths at NMSU-Alamogordo is the flexibihty of lASP to 
adjust to changing needs. 

How does L\SP fit into the SPRE Compliance Matrix? 

Dealing with the increasmg number of accountability requirements such as the SPRE 
Standards, we followed earUer successful and innovative patterns of managing changing issues. 
When increasing numbers of accountability standards became a major issue that confronted the 
college, the lASP Committee estabUshed an Outcomes Committee. The Outcomes Committee 
was tasked with the goal of creating a structured student outcome assessment program to comply 
with all external requirements. 
How do we organize all of these requirements? 

The Office of Institutional Effectiveness selected an Outcomes Assessment Coordinator to 
update the assessment matrixes that were the basis of the Developing a Process for 
Improvement Through Assessment of a Two-Year Branch Community College (Himebrook, 
Twomey, Beck, Flores, & Elliott, 1992). This report was the culmination of work done for the 
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NCA self-study. Another important tool was an AACC Special Report, Community Colleges: 
Core Indicators of Effectiveness (1994). Additional literature review was done to document 
other possible outcomes assessment approaches. A small resource library was established in the 
Office of Institutional Efifectiveness which catalogued the myriad of papers, brochures, surveys, 
and models that have been collected over the years. The resource library' is available to facult>' 
and stafiF who may need help with their assessment projects. 
What are our Mission, Goals, and Purposes? 

Searching for a way to define its goals, the Outcomes Coimnittee dealt with the following 
questions: 

1. What could we do? The dream 

2. What should we do? The accountability and reporting requirements 

3. Vvhat do we really want to do? Ignore the requirements or comply for 
relevant and meaningful purposes 

4. What can we do? Reality - money and time issue 

The Outcomes Committee established its mission: 

The mission of the Outcomes Committee of New Mexico State University' 
Alamogordo is to develop a plan that meets a number of accountability 
statidards which measure our students ' success and satisfaction with 
NMSU'A after they have left the institution along with employers, transfer 
institutions, atid community perceptions/needs of our students * education 
at NMSU'A. 

Five goals were established, each with a purpose: 

Goal One 

The first goal of the Outcomes Committee was to set up an understandable matrix that 
collapsed the myriad of standards and criterion that NMSU-A faced. The standards included the 
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NCA Criterion Three, NMSU-A's mission and purposes, core indicators outlined in the AACC 
Special Report, Community Colleges: Core IndLutors of Effectiveness (1994), and the New 
Mexico Commission on Higher Education Final Draft Standards for State Postsecondary Review 
Program- SPRE (i 994). The purpose was to determine that the outcome assessments developed 
to meet SPRE were not in conflict with the noission and purposes of NMSU-A. 

Goal Two 

The second goal of the Outcomes Committee was to research and develop ways to 
measure the effectiveness of student learning and success after the coD:q)letion of a program. We 
will measure employment success of graduates of our vocational training programs. We will 
measure the overall impact of the general education component of our degree programs and the 
resuhing satisfaction of the transfer institution. We will measure the community's perceptions of 
the institution's success. The purpose is to determine the most eflScient and effective way to 
measure satisfaction and success of NMSU-A's educational programs. 

Goal Three 

The third goal of the Outcomes Committee was to catalog every external reporting 
requirement that required some type of assessment. The purpose was to develop a comprehensive 
assessment plan that would reduce dupUcation of assessment efforts. 

Goal Four 

The fourth goal was to analyze the cost of survey instruments. This included a 
comparison of the cost of contracting student outcomes and employer assessments to an outside 
firm and the cost of developing sur\'eys and conducting student outcomes assessment in-house. 
llie purpose was to detenmne the most cost effective way to meet the goals of student outcomes 
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and employer assessments. This was very in^ortant given the limited resources at NMSU- 
Alamogordo. 

Goal FK'e 

The fifth goal of the Outcomes Committee was to research and develop an effective way 
of measuring a student's ability to complete a program (SPRE Standard Three) where 
appropriate. The purpose was to develop an institutional strategy to deal with the SPRE standard 
of a student's abiUty to successfully complete the educational program they chose. This is often 
referred to as "ability to benefit''. 
What kinds of assessments are currently being done? 

After three years of operation, the pace of faculty assessment for lASP cycle was slowed. 
Currently, the faculty evaluation document includes an assessment component. The faculty will 
design one expanded and innovative assessment project each year. The linkage of the assessment 
of student academic achievement and faculty evaluation shows the importance NMSU-A places 
on assessment. 

One goal of the Outcomes Committee was to inventory campus assessment efforts and 
needs. This effort was accompUshed with extended interviews of program and division heads. 
(See Exhibit A-1 Questions). A two-page questionnaire was developed. (See Exhibit A-2 
Assessment Survey). The purpose of the internal faculty and staff interviews was to catalog 
current assessments activities. We reviewed assessments that are required by federal and state 
licensing authorities and assessments that need to be conducted in the future. 

A major area of concern is job placement of our students. The Outcomes Assessment 
Coordinator made the assumption that many program coordinators, working closely with students 
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and employers, should have detailed job placement infonnation as required by SPRE Standard 15. 
The opportunity was taken during the interviews to determine if the program coordinators had 
access to job placement information. 

The internal faculty and stafiF interviews also allowed us to gauge faculty support of our 
student outcomes assessment program, obtain faculty suggestions on the direction an outcome 
assessment plan should take, and to prepare the faculty for changes in higher education that are 
leading to greater accountability to the community they serve. 

Evaluating the results of the internal faculty and staflF interviews indicated that 52% of the 
faculty and staff have some type of advisory board, with all but one of the program coordinators 
in the vocational-technical area having an advisory board. 57% of faculty, including most in 
vocational technical programs, were aware of the comparative costs of their instructional area. 
SPRE Standards require related job placement information, local job market conditions, and 
statistical environmental scanning information. Information generated from the internal faculty 
and staff interviews suggests that the program advisory boards could serve as a focus group to 
provide some of this information. 
How did we Develop the SPRE Compliance Matrix? 

Initially, the development process of the SPRE Compliance Matrix centered on a literature 
review of tools that could help us summarize the various accountability standards into a more 
understandable matrix. We also studied the assessment program that was designed by an earlier 
assessment committee at NMSU-Alamogordo. This assessment program served as the foundation 
for the lASP process. Much of this work was rolled into the development of the SPRE 
Compliance Matrix. This made the development and writing process much more efficient and 
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cost effective. Exhibit B is a table listing the major categories of standards for the SPRE 
Compliance Matrix. Exhibit C-1 and C-2 is the SPRE Compliance Matrix. 
Problems in Developing the Matrix 

The problems the Outcomes Committee had during the development the SPRE 
Comphance Matrix revolved around: 

1 ) Determining the pomt in time when we need to say, "These are the standards we are gomg 
to review.'' This is necessary because the accountabiUt>' standards seem to constantly 
change. 

2) Determining which standards may become serious issues in the future. 

3) Determining the cost/benefit ratio of periodically reviewing accountability standards. 

4) Determining which standards are the most relevant for the college. 

5) Determining which institutional unit will be responsible for SPRE comphance. 
What's Left to Do? 

1 ) Usmg the SPRE Comphance Matrix as a guide, the Outcomes Conamittee needs to 
compile a matrix of appropriate outcomes for each mission and purpose of the college. We need 
to answer the followmg questions. What kind of measures and data sources do we need to 
achieve our mission and purposes and comply with other external reporting requirements? Are 
these outcomes currently being measured? Who is responsible for determining which measures 
and data sources should be used? Who implements the collection of the data? Who controls 
accumulated data? Are the measures and data sources related to the implementation schedule as 
outlined in the time Ime? How are we going to use the data? 

2) As indicated earher, an overall time lino needs to be developed and implemented. From 
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the time line, the Outcome Committee's goal of developing a plan for a structured student 
outcome assessment program to comply with all e?ctemal requirements can be jSnalized, A 
procedural manual can be developed and revised periodically to adjust to changing needs of the 
college. 

3) Since a number of the accountability standards require onsite visits from the reviewing 
agencies, the Outcomes Committee needs to look at each onsite review and design an internal 
audit system to insure compliance. TWs should lessen the preparation time for faculty and staflf 
v/hen each onsite review is conducted 
W hat's in the Future? 

1) The SPRE Compliance Matrix should be reviewed periodically to determme if there have 
been significant changes in accountability requirements. The development of the SPRE 
Compliance Matrix was based on earlier work at NMSU-Alamogordo that resulted in the lASP 
process. Each review of our mission and purposes statement should be responsive to changing 
accountability requirements. This will save time and critically limited resources. It is important 
that future reviews are cost effective and relevant to the needs of the college. 



The development of the SPRE Compliance Matrix^ building on earlier successful efforts at 
NMSU-Alamogordo, has helped NMSU-Alamogordo to determine if daunting accountability 
requirements are consistent with its mission and purposes. This matrix has served as the basis for 
development for outcomes assessment data collection instruments, research questions, 
implementation schedules, and operating procedures according to SPRE and the NMSU-A 
Integrated Assessment and Strategic Planning (lASP) Process: What We\'e Learned and 



Summary 




ERIC 



Hawkins & Lillibridge PAGE 9 

Where We're Going (Twomey, Lillibridge, Hawkins, & Reidlinger, 1994). From this matrix, an 
institutional assessment plan and subsequent procedural manual will be developed for the college. 
The purpose will be to make the assessment process eflficient and effective, as unduplicated as 
possible, and relevant to the needs of the students, faculty, and staff. The ulthnate goal is for the 
college to be accountable to the community and state it serves. 
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Exhibit A-1 

INTERNAL FACULTY & STAFF INTERVIEWS 
CARL PERKINS STUDENT OUTCOMES ASSESSMENT TASK FORCE 

CONDUCTED FALL, 1994 
QUESTIONS 

Conduct brief narrative on student outcomes assessment 
L Would you agree that we should measure our former students' satisfaction with NMSU-A? 

Yes__ No^ 

Remarks 



2. Do you have some questions related to outcome's assessment you would like to ask of your former 

students? Yes No_ 

Remarks 



3. Are you aware of the changes in higher education that are leading to greater accountabiUty and 

quantitative methods of evaluatmg eflFectiveness? Yes No_ 

Remarks 



4. Are you required to make any reports to state and/or federal authorities? Yes No_ 

Remarks 



5. Do you support a student outcome*s assessment, not only talking to students & graduates who have 
been gone from NMSU-A for several years, but local employers for their attitudes our students' success? 

Yes No_ 

Remarks 



6. What would you like to see a student outcome's assessment accomplish? 



7. Would you prefer to do a student outcome's assessment under your direction or centralized under one 

area? Centralized Decentralized_ 

Remarks 



Name 



Department 

gl^^ate 23 Interviewer 



NAME 

CAMPUS PHONE 



Exhibit A-2 
ASSESSMENT SURVEY 



L CURRENT ASSESSMENTS 

If you have collected data in any of the following areas during the past year, please put an "X" by the 
appropriate item (Data can include such things as evaluation documents, statistical faiformation, correspon- 
dence, memos, etc.) 

STUDENT PROGRESS 

A. Student Goal Attainment 

B. Student Persistence (Fall to Fall) 

C. Degree Completion Rates 

CAREER PREPARATION 

D. Student Placement Rate in the Work Force 

E. ^Employer Assessment of Students 

TRANSFER PREPARATION 

F. ^Number and Rate of Students Who Transfer 

G. Students* Performance After Transfer 

devt:lopmental education 

H. Student Success in Subsequent, Related Coursework 

1. ^Number enrolling who need remediation 

^ 2. Number needmg remediation who are served 

3. Followup comparison of achievement for students wiio undertake 

remediation with those who do not 
GENEF.AL EDUCATION 

I. ^Demonstration of Critical Literacy Skills 

J. ^Demonstration of Citizenship Skills 

CUSTOMIZED EDUCATION 
K Client Assessment o f Programs and Services 

COMMUNTTY DEVELOPMENT 

L. ^ResponsK'eness to Community Needs 

M. ^Participation Rate in Service Area 

If you check any of the above items, please pro\ide a brief summary of the nature of the data collected. 
ITEM 



ITEM 



ITEM 



M 



Please return to Institutional Research. For information and questions call Linda Hawkins, #3623, Monday- 

14 



l^j^^idnesday, 9:00 am - 3:00 pm 



n. REQUIRED ASSESSMENTS BY FEDERAUSTATE/LICENSING AUTHORITIES 

Please list any assessments you currently are doing that are required by organizations and institutions 
outside NMSU-A. 

MONTH REPORT REOUIRING 
DESCRIPTION OF ASSESSMENT DUE AUTHORITY 



Any additional information you can provide would be very helpful. 



Part of the final draft standards for Poinsecondary Institutions Review Progi am (SPRE) include detailed 
job placement infoimation including available and projected jobs, jcb market conditions, actual annual 
earnings of employers related to progi am graduates; statistical information on long-term earnings; 
tuition/fee comparisons with similar instimtions. We would appreciate your help with the following 
questions. 

A. Do you hiive an industry-related advisory hoard? Yes No 

B. Do you know what the local and regional job market conditions are for 

your instructional area inclutling the number of jobs available? Yes No 

C. Do you have know vAiiM the projected local imd regional job market 

needs are fox yoiu^ instaictional area? Yes_ No 

D. Do you know w4iat the long-t«rm earning projections are for your students 

in your program? Yes^ No 

E. Do you know wiiat the coii^arative costs fire of tuition and 

fees with similar institutions in the region for yoUi** program? Yes_^ No 

F. Do you have stai*?stical data on past performance of your program graduates such as: 

job placement rates Yes No 

graduation rates Yes No 

starting salaries Yes No 

licensing examination pass rates Yes No 

ra. PROPOSED ASSESSMENTS 

if you have suggestions for as&essmrats that you think would be useful to conduct at NMSU-A, please 

indicate those below. 
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